{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Limitation of selecting feature using a model\n",
    "\n",
    "In this notebook, we want to show a limitation when using a machine-learning\n",
    "model to make a selection.\n",
    "\n",
    "Indeed, one can inspect a model and find relative feature importances. For\n",
    "instance, the parameters `coef_` for the linear models or\n",
    "`feature_importances_` for the tree-based models carries such information.\n",
    "Therefore, this method works as far as the relative feature importances given\n",
    "by the model is sufficient to select the meaningful feature.\n",
    "\n",
    "Here, we will generate a dataset that contains a large number of random\n",
    "features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_classification\n",
    "\n",
    "data, target = make_classification(\n",
    "    n_samples=5000,\n",
    "    n_features=100,\n",
    "    n_informative=2,\n",
    "    n_redundant=5,\n",
    "    n_repeated=5,\n",
    "    class_sep=0.3,\n",
    "    random_state=0,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's build a model which will not make any features selection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "model_without_selection = RandomForestClassifier()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will evaluate this model by a k-fold cross validation and store the results\n",
    "in a pandas dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.model_selection import cross_validate\n",
    "\n",
    "cv_results_without_selection = cross_validate(\n",
    "    model_without_selection, data, target, cv=5\n",
    ")\n",
    "cv_results_without_selection = pd.DataFrame(cv_results_without_selection)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we will build another model which will include a feature selection step\n",
    "based on a random forest and evaluate it as well with cross-validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.feature_selection import SelectFromModel\n",
    "\n",
    "feature_selector = SelectFromModel(RandomForestClassifier())\n",
    "model_with_selection = make_pipeline(\n",
    "    feature_selector, RandomForestClassifier()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_results_with_selection = cross_validate(\n",
    "    model_with_selection, data, target, cv=5\n",
    ")\n",
    "cv_results_with_selection = pd.DataFrame(cv_results_with_selection)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can compare the testing score of the two models. For this matter, we are\n",
    "combining results in a single dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_results = pd.concat(\n",
    "    [cv_results_without_selection, cv_results_with_selection],\n",
    "    axis=1,\n",
    "    keys=[\"Without feature selection\", \"With feature selection\"],\n",
    ").swaplevel(axis=\"columns\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can check the testing score of each the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "color = {\"whiskers\": \"black\", \"medians\": \"black\", \"caps\": \"black\"}\n",
    "cv_results[\"test_score\"].plot.box(color=color, vert=False)\n",
    "plt.xlabel(\"Accuracy\")\n",
    "_ = plt.title(\"Limitation of using a random forest for feature selection\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model that selected a subset of feature is less performant than a random\n",
    "forest fitted on the full dataset.\n",
    "\n",
    "We can rely on some aspects tackled in the notebook presenting the model\n",
    "inspection to explain this behaviour. The decision tree's relative feature\n",
    "importance will overestimate the importance of random feature when the\n",
    "decision tree overfits the training set.\n",
    "\n",
    "Therefore, it is good to keep in mind that feature selection relies on\n",
    "procedures making some assumptions, which can be perfectible."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}